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DETAILED ACTION 
Specification 

1 . The disclosure is objected to because of the following informalities: 

On page 25, line 8, "418" should be changed to "420" to correspond to figure 7. 
Appropriate correction is required. 

2. Claim 45 is objected to because of the following informalities: 
Referring to claim 45, "is matches" should be changed to "matches". 
Appropriate correction is required. 

Claim Rejections - 35 USC §102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 351(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

4. Claims 44-46, 48-49, 53-54, 56, 77-82, 88-90, and 95-97 are rejected under 35 U.S.C. 
102(e) as being anticipated by Krumm (U.S. Patent No. 6,61 1,622). 

Referring to claim 77, which is representative of claims 44 and 88, 

i. A processor is illustrated by Krumm in figure 1 by the processing unit 21. 

ii. A memory medium coupled to the processor, wherein the memory medium stores 
color match location software is illustrated by Krumm in figure 1 by the system memory 
22, ROM 24, RAM 25, magnetic disk 29, and optical disk 31. Krumm illustrates that the 
RAM contains program data 38 in figure 1. 
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iii. Automatically determining color features of the template image is explained by 
Krumm in column 8, lines 47-51. Krumm explains that model histograms, corresponding 
to color features, are created for images of people or objects that are desired to be 
identified, corresponding to template images. 

iv. Locating one or more regions of the target image that match the color features of 
the template image is explained by Krumm in column 8, lines 58-62 and illustrated in 
figure 2 by reference number 206. Krumm explains that the matched regions are 
designated to correspond to the template image. 

Referring to claim 45, generating information specifying a location for each region of the 
target image that matches the color features of the template image is illustrated by Krumm in 
figure 2 by reference number 206. The information specifying the location for each region of the 
target image corresponds to the designator of each cell of the target image. Krumm illustrates 
that the color histograms are used for matching in figure 2 by reference number 202. 

Referring to claim 89, which is representative of claims 46 and 78, 

i. A display device is illustrated by Krumm in figure 1 by the monitor 47. 

ii. For at least one region of the target image that matches the color features of the 
template image, displaying information on a graphical user interface indicating a location 
of the region within the target image is explained by Krumm in column 8, lines 62-64. 
Krumm explains that the designated people or objects, corresponding to the regions (or 
cells) in the target image that match the color features of the template image, are 
identified in the image of the scene. 
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Referring to claim 79, which is representative of claim 48, receiving the target image 
from one of a group consisting of a memory medium, a hardware device, and a software 
application is illustrated by Krumm in figure 1 by the program data 38 and the processing unit 
21 . The processing unit receives the target image to perform the step of segmentation as 
illustrated in figure 2, reference number 200 and color matching as illustrated in figure 2, 
reference number 206, from the program data 38 stored on the RAM 25, corresponding to the 
memory medium. 

Referring to claim 49, which is representative of claim 90, either the template image or 
the target image being a portion of a larger image is explained by Krumm in column 8, lines 48- 
51 . Krumm explains the images of people or objects are used s template images, which are 
portions of a scene being captured. 

Referring to claim 80, which is representative of claims 53 and 94, 

i. Automatically performing a color characterization analysis of the template image 
is explained by Krumm in column 8, lines 47-51. Krumm explains that the model 
histogram, corresponding to the color characterization of the template image, is 
performed before the color matching is performed. 

ii. Searching for regions of the target image having a color characterization that 
matches, at least to a degree, the color characterization of the template image is explained 
by Krumm in column 8, lines 51-64. Krumm explains that the similarity degree between 
the color histograms is calculated for each region of the target scene. The degree of 
similarity is then compared to a predetermined threshold (corresponding to the "at least to 
a degree") in order to determine whether the region matches the template image. 
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Referring to claim 81, which is representative of claims 54 and 95, 

i. Performing a color characterization analysis for a plurality of regions within the 
target image to generate color characterization information for each of the target image 
regions is illustrated by Krumm in figure 2 by reference number 202. In step 200, 
Krumm illustrates that the target image is segmented into a plurality of regions. In step 
202, Krumm computes a color histogram (corresponding to a color characterization 
analysis) for each of the segmented regions. 

ii. Comparing the color characterization information of the template image with the 
color characterization information for each of the target image regions is illustrated by 
Krumm in figure 2 by reference number 204. 

iii. Determining one or more target image regions having a color characterization that 
matches, at least to a degree, the color characterization of the template image is illustrated 
by Krumm in figure 2 by reference number 206. 

Referring to claim 82, which is representative of claims 56 and 96, 

i. Examining color information of at least a subset of pixels is explained by Krumm 
in column 10, lines 33-37. Krumm explains that the color histogram is created for the 
extracted regions (corresponding to a subset of the color image pixels). 

ii. Assigning each examined pixel to a color category that corresponds to a portion 
of a color space is explained by Krumm in column 10, line 63 to column 11, line 6. Each 
color space region defined by Krumm corresponds to a portion of the RGB color space. 

iii. Determining information indicative of the allocation of the examined pixels 
across color categories is explained by Krumm in column 11, lines 17-23. The 
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information indicative of the allocation of the examined pixels is the count of pixels 
belonging to each category. 

iv. Comparing the information obtained in the color characterization analysis of the 
region to the information obtained in the color characterization analysis of the template 
image in order to determine whether the region has a color characterization that matches, 
at least to a degree, the color characterization of the template image is explained by 
Krumm in column 1 1, line 49 to column 12, line 5. Krumm illustrates that once the 
maximum degree of similarity is determined in figure 6, step 602, it is compared to a 
predetermined threshold in step 604 to determine if the extracted region matches the 
template. 

Referring to claim 97, 

i. Performing a color characterization of a template image is illustrated by Krumm 
in figure 2 by reference number 204. The color characterization analysis of the template 
image corresponds to the created model histograms associated with people or objects. 

ii. Searching for regions of the target image having a color characterization that 
matches, at least to a degree, the color characterization of the template image is explained 
by Krumm in column 8, lines 51-64 and illustrated in figure 2 by reference numbers 204 
and 206. Krumm explains that the similarity degree between the color histograms is 
calculated for each region of the target scene. The degree of similarity is then compared 
to a predetermined threshold (corresponding to the "at least to a degree") in order to 
determine whether the region matches the template image. 
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iii. Performing a color characterization analysis for a plurality of regions of the target 
image is illustrated by Krumm in figure 2 by reference number 202. 

iv. Examining color information of at least a subset of pixels in order to perform 
color characterization analysis for the template image and each of the plurality of regions 
of the target image is explained by Krumm in column 1 1 , lines 1 7-23. The same 
processing is performed on the template image in order to create the previously created 
model histograms associated with people or objects". 

v. Assigning each examined pixel to a color category that corresponds to a portion 
of a color space is explained by Krumm in column 10, line 63 to column 11, line 6. Each 
color space region defined by Krumm corresponds to a portion of the RGB color space. 

vi. Determining information indicative of the allocation of the examined pixels 
across color categories is explained by Krumm in column 11, lines 17-23. The 
information indicative of the allocation of the examined pixels is the count of pixels 
belonging to each category. 

vii. Comparing the information obtained in the color characterization analysis of the 
region to the information obtained in the color characterization analysis of the template 
image in order to determine whether the region has a color characterization that matches, 
at least to a degree, the color characterization of the template image is explained by 
Krumm in column 11, line 49 to column 12, line 5. Krumm illustrates that once the 
maximum degree of similarity is determined in figure 6, step 602, it is compared to a 
predetermined threshold in step 604 to determine if the extracted region matches the 
template. 
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Claim Rejections - 35 USC § 103 



5. 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 



obviousness rejections set forth in this Office action: 



(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 



6. Claim 47 is rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in view 
of Tao (U.S. Patent No. 5,799,105). 

Referring to claim 47, for at least one region of the target image that matches the color 
features of the template image, displaying information on a graphical user interface indicating a 
degree to which color information of the region matches color information of the template image 
is not explicitly explained by Krumm. However, Tao illustrate in figure 8, reference number 819 
that the grade assignment of each object is output on a display. The grade determination is 
dependent of the degree of matching of color information between a template object 
(corresponding to the grade of fruit) and the target object (the object being imaged) as illustrated 
by Tao in figure 8, reference numbers 815 and 817. By displaying the degree of similarity (or 
matching) between a template image and a target image, a user may intervene in the color 
matching system of Krumm and Tao, thereby reducing the probability of true rejections. 
Therefore, it would have been obvious to one of ordinary skill in the art at the time the invention 
was made to display information on a graphical user interface indicating a degree to which color 
information of the region matches color information of the template image, as suggested by Tao, 
in the color matching system of Krumm, because the color matching accuracy will be improved. 
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7. Claims 51-52, 67-70, 86-87, and 92-93 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Krumm in view of Hsu et al (U.S. Patent No. 6,078,701). 

Referring to claim 86, which is representative of claims 51 and 92, performing multiple 
search passes through the target image according to a coarse-to-fine search heuristic is not 
explicitly explained by Krumm. However, Hsu et al illustrate color matching a template image 
through a target image using a coarse-to-fine heuristic in figure 3 by reference numbers 304 and 
306. Hsu et al use the coarse- to-fine heuristic in order to color match regions of two separate 
images so that the image can be accurately stitched together as explained in column 4, lines 34- 
47. Therefore, it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to use the coarse-to-fine heuristic, as suggested by Hsu et al, in order to 
determine portions of a target image that match a template image in the system of Krumm 
because the coarse-to-fine heuristic is well-known in the art to provide faster image matching, 
resulting in a more efficient color match system. 

Referring to claim 87, which is representative of claims 52 and 93, 

i. Performing a first-pass search through the target image to find initial match 
candidate areas is not explicitly explained by Krumm. However, Hsu et al illustrate a 
coarse registration module in figure 3 by reference number 304. 

ii. Performing one or more subsequent search passes in which proximal regions 
proximal to the candidate areas are search in order to find a best-matching region in the 
proximal region is not explicitly explained by Krumm. However, Hsu et al illustrate a 
fine search in figure 3 by reference number 306. Hsu et al explain in column 5, lines 7- 
30 that the coarse and fine search steps validate hypothesis of neighbor matches. Hsu et 



# 
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al then explain that the search is conducted until a predefined level of topological and 
alignment accuracy is achieved in column 5, lines 31-42. Therefore, it would have been 
obvious to one of ordinary skill in the art at the time the invention was made to perform a 
first-pass and a second-pass through a target image to find accurate matching area 
because the multiple pass (or coarse-to-fine) search is well-known in the art to provide 
faster image matching, resulting in a more efficient color match system. 
Referring to claim 67, 

i. Performing a first-pass search through the target image in order to find color 
match areas corresponds to claim 87i. 

ii. For each color match candidate area found in the first-pass search, searching a 
proximal region proximal to the color match candidate area in order to find a best- 
matching region in the proximal region corresponds to claim 87ii. 

Referring to claim 68, 

i. Determining a plurality of sample regions at which to sample the color 
information of the target image is illustrated by Krumm in figure 2 by reference numbers 
200 and 202. 

ii. For each of the plurality of sample regions, determining a measure of difference 
between the color information of the sample region and the color information of the 
template image is illustrated by Krumm in figure 2 by reference number 204. 

iii. For each of the plurality of sample regions, designating the sample region as a 
color match candidate area if the measure of difference between the color information of 
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the sample region and the color information of the template image is smaller than a 
threshold value is illustrated by Krumm in figure 2 by reference number 206. 
Referring to claim 69, 

i. Performing a color characterization analysis of the template image is illustrated 
by Krumm in figure 2 by reference number 204. The color characterization analysis of 
the template image corresponds to the created model histograms associated with people 
or objects. 

ii. For each of the plurality of sample regions, performing a color characterization 
analysis for the sample region is illustrated by Krumm in figure 3 by reference number 
202. 

iii. Determining the measure of difference between the color information of each 
sample region and the color information of the template image comprises comparing 
information obtained in the color characterization analysis of the sample region with 
information obtained in the color characterization of the template image is illustrated by 
Krumm in figure 2 by reference number 204. The color histograms of the regions of the 
target image are compared with the color histograms of the template image. 
Referring to claim 70, 

i. The template image and the target image each comprising a plurality of pixels is 
explained by Krumm in column 11, lines 17-23. 

ii. Examining color information of at least a subset of pixels is explained by Krumm 
in column 10, lines 33-37. Krumm explains that the color histogram is created for the 
extracted regions (corresponding to a subset of the color image pixels). 
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iii. Assigning each examined pixel to a color category that corresponds to a portion 
of a color space is explained by Krumm in column 10, line 63 to column 11, line 6. Each 
color space region defined by Krumm corresponds to a portion of the RGB color space. 

iv. Determining information indicative of the allocation of the examined pixels 
across color categories is explained by Krumm in column 1 1, lines 17-23. The 
information indicative of the allocation of the examined pixels is the count of pixels 
belonging to each category. 

v. Comparing the information obtained in the color characterization analysis of the 
region to the information obtained in the color characterization analysis of the template 
image comprises comparing the allocation of the examined pixels across color categories 
for the sample region and the template image is explained by Krumm in column 11, lines 
49-64. 

8. Claims 57-58 are rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in 
view of Kato et al (U.S. Patent No. 6,665,446). 
Referring to claim 57, 

i. Determining a sub-sampling size is not explicitly explained by Krumm. 
However, Kato et al illustrate sub-sampling four pixels into a single pixel in figure 3. 

ii. Using the sub-sampling size to determine the size of at least a subset of pixels 
examined for each of the plurality of regions of the target image is not explicitly 
explained by Krumm. However, Kato et al illustrate a sub-sampling size of four in figure 
3. The sub-sampling method of Kato et al can be used in the system of Krumm in order 
to reduce the information in each region of the target image. Therefore, it would have 
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been obvious to one of ordinary skill in the art at the time the invention was made to use 
determining a sub-sampling size and use the sub-sampling size to determine the size of at 
least a subset of pixels examined for each of the plurality of regions of the target image 
because the information in each region of the target image in the system of Krumm 
would be reduced, reducing the computational power and memory required. 
Referring to claim 58, 

i. Examining color information for each pixel in the template image is illustrated by 
Krumm in figure 2 by reference number 204. Each pixel of the template image is 
processing in order to create the "previously created model histogram associated with 
people or objects". 

ii. Examining color information of only a subset of the pixels in a region of the 
target image is not explicitly explained by Krumm. However, Kato et al illustrate a sub- 
sampling size of four in figure 3. The sub-sampling method of Kato et al can be used in 
the system of Krumm in order to reduce the processing required to determine the color 
characterization of each region of the target image. Therefore, it would have been 
obvious to one of ordinary skill in the art at the time the invention was made to examine 
color information of only a subset of the pixels in a region of the target image because the 
information in each region of the target image in the system of Krumm would be reduced, 
reducing the computational power and memory required. 

9. Claim 63 is rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm. 

Referring to claim 63, which is representative of claim 84, for each color category of the 
color space, comparing the percentage of template image pixels assigned to the color category to 
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the percentage of target image region pixels assigned to the color category is not explicitly 
explained by Krumm. However, Krumm explains in column 11, lines 49-64 that for each color 
category, the number of pixels present from the template image and the number of pixels present 
from the region of the target image are compared. Krumm further explains a normalization 
procedure in column 12, liens 17-24, wherein the degree of similarity (or count) is converted to a 
similarity percentage between 0 and 1 . The same normalization procedure can be utilized by 
Krumm in order to determine the percentage of pixels present in each color category rather than 
the number of pixels. The effect of such as alteration would not change the system of Krumm, as 
the degree of similarity will be output as a percentage, rather than it being converted to a 
percentage at a later step. Therefore, it would have been an obvious matter of design choice to 
one of ordinary skill in the art at the time he invention was made to modify the system of Krumm 
by normalizing the number of pixels present in each category before the degree of similarity is 
determined rather than normalizing the degree of similarity later, since the Applicant has not 
disclosed that normalizing the number of pixels present in each category solves any stated 
problem and it appears that the normalization of the degree of similarity as disclosed by Krumm 
would perform equally as well. 

10. Claim 55 is rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in view 
of Park et al (U.S. Patent No. 6,584,212). 
Referring to claim 55, 

i. Determining a step size is not explicitly explained by Krumm. However, Park et 
al illustrate a first step size of two pixels in figure 1 by the two vectors originating from 
coordinate (0,0). 
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ii. Determining locations for the plurality of regions within the target image for 
which the color characterization analysis is performed is illustrated by Krumm in figure 2 
by reference numbers 200 and 202. Krumm explains determining the regions in column 
9, line 53 to column 10, line 30. 

iii. The step size being used in determining locations for the plurality of regions 
within the target image is not explicitly explained by Krumm. However, Park et al 
explain in column 1, lines 49-52 that a difference value is determined at every search 
location. Park et al explain in column 1, lines 11-18 that the multiple step search sizes 
reduce the calculations required to determine the best match in a target image. Therefore, 
it would have been obvious to one of ordinary skill in the art at the time the invention was 
made to use a step size in order to determine locations for the plurality of regions within 
the target image used by the system of Krumm because the region selection process 
would be more efficient. 

11. Claims 50, 59-62, 83, 85, and 91 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Krumm in view of Chen et al (K. Chen et al, "Similarity-Based Retrieval of 
Images Using Color Histograms", Proceedings of the SPEE, 26 January 1999, XP000983710, 
chapters 2,3). 

Referring to claim 83, which is representative of claim 59, 

i. The template image and the target image comprising hue, saturation, and intensity 
(HSI) color information is explained by Krumm in column 10, lines 38-49. Krumm 
explicitly explains that the color images may contain HIS color information. 
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ii. Examining HSI information of least a subset of pixels is explained by Krumm in 
column 10, lines 38-49. Krumm explicitly explains that the HIS information may be 
examined for the template image and the segments of the target image. 

iii. Assigning each examined pixel to a color category that corresponds to a portion 
of HSI color space is not explicitly explained by Krumm. Krumm does explain color 
categories corresponding to the RGB color space in column 10, line 63 to column 11, line 
6. Krumm further explains that the HSI color characterization may be used instead of the 
RGB characterization in column 10, lines 44-49. However, Chen et al explain that color 
categories are created according to the HSI color space on page 644, paragraphs 5 and 6 
(Q2 and Q3). Chen et al explain that the pixels of an image are assigned to the color 
categories on page 644, paragraph 2. Chen et al explain that color categories are assigned 
to portions of the HSI color space in order to reduce the number of histogram color bins, 
thereby reducing the processing required to determine the degree of similarity between 
the template image and the target image regions. Therefore, it would have been obvious 
to one of ordinary skill in the art at the time the invention was made to assign each 
examined pixel to a color category that corresponds to a portion of HSI color space, as 
explained by Chen et al, in the system of Krumm because Krumm suggests the HSI color 
space as an alternative to the RGB color space and the color categories reduce the 
processing required in the system of Krumm. 

Referring to claim 85, determining contributions which a pixel should make to a plurality 
of color categories and distributing the weight of the pixel across the plurality of color categories 
in accordance with the determined contributions is not explicitly explained by Krumm. 
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However, Chen et al illustrate overlapping membership functions for soft-decision 
histogramming in figure lb. The weight of the pixel is distributed due to the overlapping of the 
membership functions, wherein a value for "c" (on the x-axis) corresponds to multiple color 
categories. The values of the membership functions at "c" determine the weights of the pixel in 
each category. Chen et al explain on page 644, paragraph 7, that soft-decision histogramming is 
an improvement over hard-decision histogramming, as used in the system of Krumm, because 
small, random color variations will not render different histograms. Therefore, it would have 
been obvious to one of ordinary skill in the art at the time the invention was made to use soft- 
decision histogramming, as suggested by Chen et al, over hard-decision histogramming in the 
system of Krumm because soft-decision histogramming provides more accurate and consistent 
representations of images exhibiting small, random color variations. 

Referring to claim 60, converting either of the template image or the target image to HSI 
format is explained by Krumm in column 10, lines 44-49. Krum explains using the HSI color 
space instead of the RBG color space, which inherently requires conversion from one color space 
to the other. 

Referring to claim 61, 

i. Determining if the examined pixel can be categorized as either black, gray, or 
white based on one or more of saturation and intensity values of the respective pixel is 
not explicitly explained by Krumm. However, Chen et al explain on page 644, paragraph 
6 (Q3) that the intensity of each pixel is divided into five categories, black, dark, gray, 
silver, and white. 
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ii. Assigning the examined pixel to a black, gray, or white category if the examined 
pixel can be categorized as black, gray, or white is not explicitly explained by Krumm. 
However, Chen et al explain that the pixels of an image are assigned to the color 
categories on page 644, paragraph 2. The color categories in the Q3 quantization scheme 
include black, gray, and white, corresponding to part i of this claim above. 

iii. Determining a color category for the examined pixel based on one or more of hue 
and saturation values of the examined pixel, if the examined pixel cannot be categorized 
as black, gray, or white is not explicitly explained by Krumm. However, Chen et al 
explain on page 644, paragraph 6 (Q3) that if the pixel falls into the three intermediate 
intensity levels, then the pixels are further classified into two saturation levels and six hue 
levels. Chen et al explain that color categories are assigned to portions of the HSI color 
space in order to reduce the number of histogram color bins, thereby reducing the 
processing required to determine the degree of similarity between the template image and 
the target image regions on page 644, paragraph 2. Chen et al also explain that only the 
middle intensity regions are further classified because the human perception is more 
sensitive in those areas, thereby further reducing the number of classifications possible. 
Therefore, it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to determine and assign an examined pixel to either a black, gray or 
white pixel and determine a color category for the examined pixel if the examined pixel 
is not either black, gray, or white, as explained by Chen et al, in the system of Krumm 
because Krumm suggests the HSI color space as an alternative to the RGB color space 
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and the efficient allocation of color categories reduces the processing required in the 
system of Krumm. 

Referring to claim 62, receiving user input specifying a desired color sensitivity level to 
use in locating target image regions that match the template image and the user input 
determining a number of categories into which the color space is divided is not explicitly 
explained by Krumm. Krumm does illustrate a serial port interface with which a user can input 
data in figure 1 by reference number 46. However, Chen et al explain on page 644, paragraphs 
4-6 that 3 distinct methods of color space quantization are available, wherein each method 
utilizes a different number of color categories. It would be inherent in the system of Chen et al 
that a user would have to select which type of quantization to use, since the system has all three 
available. The user could input his/her selection through the serial port interface in the system of 
Krumm. By allowing the user to customize and optimize the search algorithm, the accuracy 
and/or speed of the search may be improved. Therefore, it would have been obvious to one of 
ordinary skill in that art at the time the invention was made to receive a user input specifying a 
desired color sensitivity level, wherein the sensitivity level adjusts the number of color 
categories, to locate target image regions that match a template region because the accuracy 
and/or speed of the search may be customized and improved. 

Referring to claim 91, which is representative of claim 50, receiving user input specifying 
search criteria to use in searching through the target image and the user input determining one or 
more parameters affecting the search through the target image is not explicitly explained by 
Krumm. However, Chen et al propose multiple quantization algorithms in order to determine 
color categories on page 644, paragraphs 4-6 (Qi, Q 2 , Q3). Chen et al further propose three 
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distinct difference metrics on page 648 by 4.2.1, 4.2.2, and 4.2.3. In order to select the desired 
quantization algorithm and distance metric, it would be inherent that a user would have to input a 
criteria into a system, such as the system illustrated by Krumm in 1. By allowing the user to 
customize and optimize the search algorithm, the accuracy and/or speed of the search may be 
improved. Therefore, it would have been obvious to one of ordinary skill in that art at the time 
the invention was made for a user to specify search criteria to determine one or more search 
parameters because the accuracy and/or speed of the search may be customized and improved. 
12. Claims 64-65 are rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in 
view of Nelson et al (U.S. Patent No. 6,243,713). 

Referring to claim 64, determining one or more dominant color categories for the 
template image, wherein one or more dominant categories are assigned a relatively larger 
proportion of examined pixels, with respect to other categories of the color space is not explicitly 
explained by Krumm. However, Nelson et al explain in column 1 1 , lines 54-64 that only the top 
Nth most significant bins are retained in the color histogram of an image. By determining the 
most dominant color categories for a template image, the amount of data being retained for each 
image is reduced, thereby reducing the computational power and memory required to store the 
characterization of the image. Therefore, it would have been obvious to one of ordinary skill in 
the art at the time the invention was made to determine one or more dominant color categories 
for the template image, from the color histogram created in the system of Krumm because the 
computational power and memory requirement for the image matching system would be 
reduced. 
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Referring to claim 65, for each dominant color category, comparing the percentage of 
template image pixels assigned to the dominant color category to the percentage of target image 
region pixels assigned to that color category is not explicitly explained by Krumm. However, 
Nelson et al explain determining the dominant color categories in column 1 1, lines 54-64. 
Nelson et al also illustrate filtering and normalizing the tokens (i.e. categories) in figure 6 by 
reference numbers 630. The normalization of the categories determines the percentage of pixels 
present in each color category rather than the number of pixels. Nelson et al explain in the 
abstract that characterizations of multimedia (including image) works are created in order to 
retrieve multimedia from a database. Therefore, it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to compare the percentage of template image 
pixels assigned to the dominant color category to the percentage of target image region pixels 
assigned to that color category because comparing the percentages of only the dominant 
categories reduces the computational power and memory requirement in the system of Krumm. 
13. Claim 66 is rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in view 
of Cymbalski (U.S. Patent No. 5,867,594). 

Referring to claim 66, performing a smoothing operation after assigning each examined 
pixel to a color category, wherein the smoothing operation comprising redistributing a portion of 
the pixels assigned to the respective color category to one or more neighboring color categories 
for each respective color category of a subset of the possible color categories is not explicitly 
explained by Krumm. However, Cymbalski explain in column 9, line 55 to column 10, line 5 
that the color categories may be smoothed by re-distributing pixels from one color category into 
another color category. Cymbalski explain that such a smoothing is performed in order to 
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simplify the histogram of the image, thereby reducing the computational power and memory 
requirement. Therefore, it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to smooth the color histogram of the system of Krumm by redistributing 
a portion of the pixels assigned to the respective color category to one or more neighboring color 
categories for each respective color category of a subset of the possible color categories because 
the histogram would be simplified, reducing the computational power and memory requirement 
in the system of Krumm. 

14. Claim 71 is rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in view 
of Hsu et al, and further in view of Nelson et al. 
Referring to claim 71, 

i, The color characterization analysis performed for the template image further 
comprising determining one or more dominant color categories, wherein one or more 
dominant categories are assigned a relatively larger proportion of examined pixels, with 
respect to other categories of the color space is not explicitly explained by Krumm or Hsu 
et al. However, Nelson et al explain in column 11, lines 54-64 that only the top Nth most 
significant bins are retained in the color histogram of an image. 

ii. Comparing information obtained in the color characterization analysis of the 
sample region with information obtained in the color characterization analysis of the 
template image comprising comparing the dominant color categories of the sample region 
and the template region is not explicitly explained by Krumm or Hsu et al. However, 
Nelson et al explain in the abstract that characterizations of multimedia (including image) 
works are created in order to retrieve multimedia from a database. By determining and 
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comparing the most dominant color categories of a template image, the amount of data 
being retained for each image would be reduced. Therefore, it would have been obvious 
to one of ordinary skill in the art at the time the invention was made to determine one or 
more dominant color categories for the template image from the color histogram created 
in the system of Krumm, and compare the dominant regions of the color characterization 
because the computational power and memory requirement for the image matching 
system would be reduced. 

15. Claim 73 is rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in view 

of Hsu et al, and further in view of Chen et al. 
Referring to claim 73, 

i. The template image and the target image comprising hue, saturation, and intensity 
(HSI) color information is explained by Krumm in column 10, lines 38-49. Krumm 
explicitly explains that the color images may contain HIS color information. 

ii. Examining HSI information of least a subset of pixels is explained by Krumm in 
column 10, lines 38-49. Krumm explicitly explains that the HIS information may be 
examined for the template image and the segments of the target image. 

iii. Assigning each examined pixel to a color category that corresponds to a portion 
of HSI color space is not explicitly explained by Krumm or Hsu et al. Krumm does 
explain color categories corresponding to the RGB color space in column 10, line 63 to 
column 11, line 6. Krumm further explains that the HSI color characterization may be 
used instead of the RGB characterization in column 10, lines 44-49. However, Chen et al 
explain that color categories are created according to the HSI color space on page 644, 
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paragraphs 5 and 6 (Q 2 and Q3). Chen et al explain that the pixels of an image are 
assigned to the color categories on page 644, paragraph 2. Chen et al explain that color 
categories are assigned to portions of the HSI color space in order to reduce the number 
of histogram color bins, thereby reducing the processing required to determine the degree 
of similarity between the template image and the target image regions. Therefore, it 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made to assign each examined pixel to a color category that corresponds to a portion of 
HSI color space, as explained by Chen et al, in the system of Krumm and Hsu et al 
because Krumm suggests the HSI color space as an alternative to the RGB color space 
and the color categories reduce the processing required in the system of Krumm and Hsu 
et al. 

16. Claim 72 is rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in view 
of Hsu et al, and further in view of Kato et al. 
Referring to claim 72, 

i. Examining color information for each pixel in the template image is illustrated by 
Krumm in figure 2 by reference number 204. Each pixel of the template image is 
processing in order to create the "previously created model histogram associated with 
people or objects". 

ii. Examining color information of only a subset of the pixels in a region of the 
target image is not explicitly explained by Krumm or Hsu et al. However, Kato et al 
illustrate a sub-sampling size of four in figure 3. The sub-sampling method of Kato et al 
can be used in the system of Krumm and Hsu et al in order to reduce the processing 
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required to determine the color characterization of each region of the target image. 
Therefore, it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to examine color information of only a subset of the pixels in a 
region of the target image because the information in each region of the target image in 
the system of Krumm and Hsu et al would be reduced, reducing the computational power 
and memory required. 

17. Claims 74-76 are rejected under 35 U.S.C. 103(a) as being unpatentable over Krumm in 
view of Hsu et al, and further in view of Park et al. 
Referring to claim 74, 

i. Determining a first step size to use in the performing the first-pass search through 
the target image is not explicitly explained by Krumm or Hsu et al. Hsu et al do illustrate 
a coarse search in figure 3 by reference number 304, but do not explicitly explain that a 
first step size is used. However, Park et al illustrate a first step size of two pixels in 
figure 1 by the two vectors originating from coordinate (0,0). 

ii. Determining a second step size, wherein the second step size is smaller than the 
first step size is not explicitly explained by Krumm or Hsu et al. Hsu et al do illustrate a 
fine search in figure 3 by reference number 306, but do not explicitly explain that a 
second step size is used. However, Park et al illustrate a second step size of one pixel in 
figure 1 by the two vectors originating from (4, -6) and (-6, 6). The second step size is 
one, which is smaller than the first step size of two. 

iii. Searching the proximal regions proximal to the color match candidate areas using 
the second step size is not explicitly explained by Krumm or Hsu et al. However, Park et 
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al illustrate in figure 1 that the proximal regions to coordinates (4, -6) and (-6, 6) are 
searched using the second step size. Park et al explain in column 1, lines 11-18 that the 
multiple step search sizes reduce the calculations required to determine the best match in 
a target image. Therefore, it would have been obvious to one of ordinary skill in the art 
at the time the invention was made to search the proximal regions proximal to the color 
match candidate areas using the second step size in order to determine matches between a 
plurality of regions within the target image and a template image in the system of Krumm 
because the region selection process would be more efficient. 
Referring to claim 75, 

i. Determining a plurality of sample regions in the proximal region at which to 
sample the color information of the target image is not explicitly explained by Krumm or 
Hsu et al. Krumm does illustrate that color information is sampled at every target region 
in figure 2 by reference number 202. However, Park et al illustrate eight sample regions 
proximal to the coordinate (0,0) in figure 1 . 

ii. Determining a measure of difference between the color information of the sample 
region and the color information of the template image for each of the plurality of sample 
regions in the proximal region is not explicitly explained by Krum or Hsu et al. The 
system of Krumm uses color information in order to determine the difference between a 
template image and regions of a target image as illustrated in figure 2 by reference 
number 204. However, Park et al explain in column 1, lines 49-52 that a difference 
measure is determined at each of the proximal search regions. 



Application/Control Number: 09/639,420 Page 27 

Art Unit: 2621 

iii. The best-matching region in the proximal region being a sample region with a 
smallest measure of difference is not explicitly explained by Krumm or Hsu et al. 
However, Park et al explain in column 1, lines 49-52 that the minimum block distance 
measure is determined from one of the center search location and eight proximal search 
locations. Park et al explain in column 1, lines 11-18 that the four-step search reduces the 
calculations required to determine the best match in a target image. Therefore, it would 
have been obvious to one of ordinary skill in the art at the time the invention was made to 
determine a plurality of sample regions at which to sample color information, determine a 
best-matching region in the proximal region with the smallest measure of difference 
between the color information so that the color matching system of Krumm and Hsu et al 
would be more efficient. 
Referring to claim 76, 

i. For each best-matching region found, determining a measure of difference 
between the color information of the region and the color information of the template 
image is illustrated by Krumm in figure 2 by reference number 204. 

ii. Designating at least a subset of the best-matching regions as the final match 
regions is not explicitly explained by Krumm or Hsu et al. However, Park et al illustrate 
in figure 1 that the best match regions correspond to the regions at coordinates (-5, 7) and 
(3, -7). By designating only the best search regions in the final step as the final match 
regions, less computational power and memory would be required to store and process 
the search regions. Therefore, it would have been obvious to one of ordinary skill in the 
art at the time the invention was made to Designate at least a subset of the best-matching 
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regions as the final match regions in the system of Krumm so that the computational 
power and memory required in the system of Krumm would be reduced, 
iii. The each best matching region designated as a final match region, the measure of 
difference between the color information of the final match region and the color 
information of the template image being less than a threshold is illustrated by Krumm in 
figure 2, by reference number 206. 

Conclusion 

18. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Tanaka (U.S Patent No. 6,519,360) - To exhibit color matching using color 
histograms as explained in the abstract. 

Wang et al (U.S. Patent No. 5,802,361) - To exhibit a low level analyzer and a 
high level analyzer as illustrated in figure 2b. 

19. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Hussein Akhavannik whose telephone number is (703)306-4049. 
The examiner can normally be reached on M-F 8:30-5:00. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo H. Boudreau can be reached on (703)305-4706. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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